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Abstract 


This  report  contains  three  independent  but  closely  related  papers  which  were  written  to 
address  a  request  by  Mr.  Walter  Hollis,  Deputy  Undersecretary  of  the  Army,  Operations 
ResearcL  Each  paper  was  presented  at  a  symposium  in  FY96  and  was  published  in  the  respective 
symposium  proceedings:  1]  “A  V/L  Taxonomy  for  Analyzing  Ballistic  Live-Fire  Events,”  46th 
Annual  Bomb  and  Warhead  Technical  Symposium,  held  at  Naval  Postgraduate  School,  Monterey, 
CA,  May  13-15,  1996;  2]  “Modeling  Ballistic  Live-Fire  Events,”  7th  Annual  TARDEC 
Symposium,  held  at  Naval  Postgraduate  School,  Monterey,  CA,  March  26-28,  1996;  and 
3]  “Developments  in  Modeling  Ballistic  Live-Fire  Events,”  16th  International  Symposium  on 
Ballistics,  held  in  San  Francisco,  CA,  September  23-28,  1996.  In  combination,  this  trilogy 
documents  the  current  state-of-the-art  in  Army  capabilities  and  specifies  the  future  directions  for 
modeling  live-foe  events  in  support  of  requirements  specified  in  United  States  Code,  Title  10, 
Section  2366,  Chapter  139,  National  Defense  Authorization  Act  for  FY87,  “Live  Fire  Testing.” 
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ABSTRACT 

Beginning  with  World  War  n  and  its  aftermath,  the  area  of  ballistic  vulnerability/lethality 
(V/L)  was  first  defined  as  a  specific  discipline  within  the  field  of  ballistics.  As  the  field  devel¬ 
oped,  various  practices  and  metrics  emerged.  In  some  cases  metrics  were  developed  which  were 
abstractly  useful  but,  for  example,  bore  no  direct  relationship  to  field  observables.  In  the  last 
decade,  as  the  issues  of  live-fire  strategies  and  model  W&A  have  gained  importance,  increased 
attention  has  been  focussed  on  V/L  with  the  intent  of  bringing  greater  rigor  and  clarity  to  the  dis¬ 
cipline.  In  part  this  effort  has  taken  the  form  of  defining  a  VIL  Taxonomy^  which,  in  essence,  is  a 
method  of  decomposing  a  series  of  concatenated  complex  processes  into  separable,  less-complex 
ones,  each  with  certain  properties  and  relationships,  one  to  another.  This  paper  attempts  to  sum¬ 
marize  these  efforts  and  illuminate  their  relevance  to  the  V/L  endgame  activities  so  critical  to 
modem  weapons  analyses. 


1.  Purpose 

Insight  into  the  processes  of  vulnerability  and  lethality  can  be  gained  through  use  of  what  is  now  called  the  Vulnera¬ 
bility!  Lethality  (V/L)  Taxonomy!^  It  was  first  generated^  as  a  by-product  of  a  program  to  improve  the  quality  of  Live- 
Fire  (LF)  Abrams  vulnerability  modeling.  In  essence,  the  V/L  Taxonomy  provides  a  method  to  decompose  the  ele¬ 
ments  of  V/L  into  a  sequence  of  simpler  constituent  parts.  As  we  will  see,  the  parts  relate  to  each  other  in  a  specific 
processing  order,  but  are  fundamentally  different,  one  from  the  other,  and  each  has  its  unique  and  appropriate  use  in 
the  general  scheme  of  V/L  assessment. 

2.  V/L  Taxonomy  via  a  Combat  Analogue 

The  V/L  Taxonomy  can  probably  best  be  introduced  via  a  description  in  terms  of  its  physical  and  engineering  pro¬ 
cesses.  Figure  1  illustrates  such  a  view  of  the  Taxonomy.  The  process  structure  is  illustrated  with  a  missile  attack¬ 
ing  an  aircraft,  although  this  process  is  applicable  for  any  ballistic  threat  against  any  target.^  The  cartoons  at  the  cen¬ 
ter  of  Fig.  1  represent  alternately  Levels^  indicated  on  the  left,  connected  by  Transformation  processes,  indicated  on 
the  right. 

We  start  with  an  explanation  of  the  process  of  Vulnerability  using  Fig.  1.  In  conventional  vulnerability,  it  is  normal 
practice  to  assume  a  warhead  hit  (or  ballistic  interaction)  with  a  target  as  a  given.  It  is  useful  to  think  in  this  context 
of  a  LF  test.  Level  1]  represents  a  complete  geometric  and  material  description  of  a  threat  (here  a  missile),  a  target 

oo  See  Appendix  A  for  standard  definitions  of  Lethality,  Vulnerability,  and  Survivability. 

1 .  Paul  H,  Deitz  and  Aivars  Ozolins,  Computer  Simulations  of  the  Abrams  Live-Fire  Field  Testings  Proceedings  of  the  XXVEI  Annual  Meet¬ 
ing  of  the  Army  Operations  Research  Symposium,  12-13  October,  1988,  Ft.  Lee,  VA;  also  Ballistic  Research  Laboratory  Memoran¬ 
dum  Report  BRL-MR-375S,  May  1989. 

t  It  will  be  noted  later  that  this  notion  can  be  applied  as  well  to  non-ballistic  threats  such  as  Directed  Energy  and  Chemical  Weapons  analyses. 
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(here  an  aircraft),  and  die  relevant  kinematics  as  the  two  just  begin  to  interact.  The  event  may  be  signaled  by  the 
instant  an  unguided  bullet  begins  to  colUde  with  a  target  or,  as  implied  by  the  illustration,  a  fuze  triggers  an  explo¬ 
sive  mechanism  in  the  vicinity  of  the  target.  The  process  of  a  LF  test  is  to  transform  an  undamaged  target  at  Level 
1]  to  a  damaged  target  at  Level  2].  The  transformation  process  is  the  LF  event  itself,  and  is  characterized  by  all  the 
physical  mechanisms  of  destruction.  They  may  include  main  penetrators,  fragments,  blast,  shock,  fire,  fumes  and 
even  synergistic  effects.  As  a  consequence  of  the  LF  transformation  event,  target  damage  may  have  occurred  at 
Level  2].  We  choose  to  think  of  Level  2]  as  characterized  by  a  list  of  killed  components;  sometimes  this  is  called  a 
damage  vector. 

A  target  which  has  received  damage  may  likely  not  continue  to  operate  as  before  damage.  In  the  case  of  a  damaged 
aircraft  illustrated  here,  parts  of  the  control  surfaces  may  be  removed,  hydraulic  lines  severed  and  electronic  boxes 
impaired.  In  a  test  which  might  be  performed,  the  rate-of-climb  might  be  measmed  to  see  how  this  key  performance 
property  may  have  been  reduced.  The  performance  test  is  the  transformation  process  which  takes  target  damage  at 
Level  2]  and  transforms  it  to  reduced  capability  at  Level  3].  The  transformation  from  Level  2]  to  Level  3]  can  be 
thought  of  as  characterized  by  engineering  relations.  It  is  important  to  note  that  the  metrics  of  Level  1],  threat-target 
initial  conditions.  Level  2],  damaged  components,  and  Level  3],  measures-of-capability  are  all  measureable  and 
objective  metrics. 

The  capability  state  of  Level  3]  should  be  characterized  by  all  capability  measures  which  cause  a  military  platform 
to  have  military  utility  or  worth  in  a  particular  mission.  For  example,  if  the  platform  can  move,  the  metrics  might 
include  measures  of  speed  and  agility.  If  the  platform  has  a  gun,  the  metrics  might  include  time-to-acquire  a  target, 
rate-of-fiie  and  hit  dispersion. 

The  final  transformation  occurs  as  a  platform  with  reduced  measures-of-capability  is  exercised  in  a  particular  mis¬ 
sion  scenario.  If  the  particular  reduced  measures-of-capability  are  unimportant  to  the  mission  at  hand,  then  the  util¬ 
ity  of  the  platform  may  remain  high.  If  not,  the  utility  may  be  reduced,  even  drop  to  zero.  The  notion  of  measures- 
of-effectiveness  or  utility  is  illustrated  as  a  Level  4]  metric  and  would  be  reached  through  an  operational  test  or  war 
experience.  Given  the  complexity  of  this  transformation  and  the  lack  of  real-world  repeatability,  we  claim  that  Level 
4]  metrics  are  essentially  not  observable,  but  rather  must  be  inferred  through  war  games  or  developed  via  subjective 
processes. 

One  of  the  key  insights  provided  by  the  V/L  Taxonomy  is  that  the  discipline  of  vulnerability  ranges  over  three  dis¬ 
tinct  kinds  of  metrics,  damage,  capability  and  utility,  and  great  care  must  be  exercised  to  see  that  these  metrics  are 
not  confused,  incorrectly  calculated  or  improperly  applied. 

In  contrast  to  vulnerabihty,  in  which  a  threat  interaction  with  a  target  is  normally  assumed,  lethality  often  includes 
the  process  of  getting  the  threat  to  the  target.  This  is  generally  true  in  the  assessment  of  direct-fire  weapons  such  as 
tank-fired  rounds.  By  contrast,  studies  of  indirect-fire  weapons,  such  as  warheads  delivered  by  artillery  or  rockets, 
generally  begin  with  warhead  initiation  in  the  neighborhood  of  the  target.  Thus  Fig.  1  includes  a  Level  0],  which 
represents  fee  initial  conditions  for  fee  launch  of  a  direct-fired  threat.  The  transformation  of  fee  threat  at  Level  0]  to 
fee  arrival  at  fee  target  (Level  1])  would  occur  here  as  fee  firing  of  fee  missile.  In  a  set  of  repeated  experiments,  a 
distribution  of  threat  arrival  conditions  could  be  generated.  One  particular  condition  at  a  time  might  be  chosen  to 
use  for  a  given  initial  threat-target  interaction  at  Level  1]. 

3.  V/L  Taxonomy  via  a  Mapping  Abstraction 

The  V/L  Taxonomy  is  useful  in  developing  fee  mathematical  abstractions  needed  for  V/L  modeling.  Each  of  fee 
levels  of  fee  process  can  be  thought  of  as  a  mathematical  space.  As  illustrated  in  Fig.  2,  fee  cartoons  in  fee  middle 
of  Fig.  1  describing  fee  Levels  have  been  replaced  wife  ellipses  representing  these  spaces. 

The  information  at  Levels  0]  through  4]  can  each  be  described  by  vectors  within  these  spaces,  here  represented  as 
bullets;  however,  fee  properties  of  fee  vectors  are  completely  different  from  one  space  to  fee  next.  As  mentioned 
above,  fee  metrics  of  damage,  capability  and  utility  are  not  interchangeable. 

As  noted  above,  a  LF  test  can  be  thought  of  as  a  mapping  from  Level  1]  to  Level  2].  If  fee  LF  shot  were  repeated  it 
is  likely  feat  random  physical  processes  could  lead  to  a  different  damage  vector,  thus  a  different  vector  would  result. 
LF  tests  and  modeling  efforts  have  shown^  feat  fee  outcome  for  many  LF  tests  can  exceed  w  individual  damage 
vectors.  The  high  dimensionality  of  Level  2]  space  is  at  the  core  of  the  difficulty  in  validating  V/L  models. 
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The  transformation  processes  listed  at  the  right  of  Fig.  1  have  been  reproduced  in  Fig.  2  as  well.  It  is  useful  to  think 
of  the  transformations  as  mathematical  operators.  These  operators  operate  on  information  on  one  level  to  yield 
information  at  the  next.  A  nomenclature  which  has  been  adopted  is  to  use  a  capital  O  (for  operator),  followed  by 
subscripts  indicating  the  input  and  output  levels.  Thus  the  ^  Operator  represents  the  mapping  of  the  threat  from 
launch  to  the  arrival  at  the  target.  The  Operator  represents  the  the  damage  mapping  process  of  a  LF  test.  The 
®2,3  represents  the  transformation  to  reduced  capability  of  a  target  following  damage.  And  the  ^  Oper¬ 

ator  represents  the  transformation  from  reduced  capability  to  military  utility  for  a  particular  mission  profile. 

4,  Granularity  and  the  V/L  Taxonomy 

In  addition  to  the  delineation  of  the  V/L  Levels  as  discussed  in  Section  1  and  the  nature  of  the  mapping  operators 
described  in  Section  2,  the  character  and  utility  of  a  V/L  code  is  further  determined  by  the  granularity  reflected  in  its 
levels.  For  a  particular  level,  granularity  describes  the  extent  to  which  a  metric  is  amalgamated  {Le.  integrated  into 
larger  elements)  vice  refined  {Le,  subdivided  into  smaller  elements).  The  former  tendency  is  referred  to  as  lower 
granularity  while  the  latter  is  higher  granularity. 

This  issue  may  be  illustrated  most  easily  by  discussing  its  effect  at  Level  1],  where  granularity  relates  directly  to  the 
resolution  embodied  in  the  target  description.^  Particularly  in  the  early  days  of  V/L  analysis,  target  descriptions  were 
not  highly  detailed.  Only  principal  volumes  of  the  target  were  modeled,  major  compartments  and  components,  cer¬ 
tainly  not  individual  wires  and  hydraulic  lines.  One  of  many  subjective  areas  of  judgement  that  a  V/L  analyst 
invokes  is  a  decision  as  to  what  level  of  detail  to  describe  the  geometry  of  the  target. 

Granularity  is  a  part  of  Level  2]  metrics  as  well.  As  damage  is  predicted,  it  is  typically  applied  to  the  geometry 
explicitly  described  at  Level  1].  For  example,  if  a  GPS  (Gunner’s  Primary  Sight)  is  modeled  simply  as  a  box  at 
Level  1],  there  are  only  very  limited  ways  to  predict  damage  at  Level  2]  to  yield  information  about  which  circuit  or 
optical  element  within  the  unit  might  now  be  dysfunctional.  One  such  way  might  be  via  empirically  derived  distribu¬ 
tions  of  the  associated  subcomponents  or  functionalities. 

So  too  at  Level  3],  the  capability  to  fire  a  gun  might  be  described  simply  as  a  Bernoulli  trial  (binary  outcome)  or, 
with  greater  detail  (granularity),  utilizing  specific  descriptors  of  gun  rate-of-fire,  time-to-acquire  targets,  hit  disper¬ 
sion,  etc. 

The  issue  isn’t  simply  that  each  Level  can  have  an  arbitrary  granularity,  but  that  the  granularity  of  a  particular  level 
requires  a  minimum  granularity  at  the  prior  level.  Using  the  previous  example,  the  high-resolution  description  of 
gun  performance  at  Level  3]  can  not  be  performed  without  the  support  of  a  sufficiently  detailed  damage  vector  at 
Level  2].  And  the  adequacy  of  the  damage  vector  at  Level  2]  is  in  turn  enabled  by  the  detail  of  the  geometry  at 
Level  1]. 

The  configuration  of  a  V/L  model  in  terms  of  the  granularity  invoked  must  be  based  on  the  set  of  desired  metrics 
which  it  is  to  support.  The  desired  metrics  may  typically  be  distributed  over  a  number  of  levels,  and  a  sufficiency 
check  must  be  made  to  ensure  that  the  resolution  required  at  a  given  level  is  adequately  supported  at  each  prior  level. 

5.  Insights  Provided  by  the  V/L  Taxonomy. 

There  are  a  number  of  important  aspects  of  the  taxonomy,  particularly  for  understanding  the  definitions  for  vulnera¬ 
bility,  lethality  and  survivability. 

•  The  V/L  metrics  associated  with  each  of  the  Levels  0]  through  4]  are  fundamentally  different,  one  from 
another.  That  is  to  say,  component  damage  across  a  weapon  platform  is  different  from  (potentially  dimin¬ 
ished)  capability  of  the  platform  is  different  from  the  (possible  reduction  in)  military  utility  of  the  platform. 
Both  vulnerability  of  a  platform  or  the  lethality  of  a  weapon  can  be  defined  in  any  combination  of  metrics 
from  Levels  2],  3],  and/or  4]. 

•  The  five  levels  are  sequential,  orthogonal  and  non-permutable. 

•  Modem  V/L  modeling  schemes  must  track  the  taxonomy  so  that  Level  2]  and  3]  metrics  can  be  compared 
with  the  results  of  field  tests. 


2.  Paul  H.  Deitz  and  Keith  A.  Applin,  Practices  and  Standards  in  the  Construction  ofBRL-CAD  Target  Descriptions,  Army  Research  Labo¬ 
ratory  Memorandum  Report  ARL-MR-103,  September  1993. 
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Fig.  1.  V/L  Taxonomy  iUustrated  with  physical  and  engineering  processes  in  the  center  column.  Ihe 
Levels  0]  through  4]  are  described  on  the  left.  The  transformation  processes  between  die  Levels  are 
described  on  the  right 
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F^.  2.  y/L  Taxonomy  illustrated  via  a  Mapping  Abstraction.  The  ellipses  in  the  middle  column  repre¬ 
sent  mathematical  spaces.  The  points  contained  within  represent  vectors.  The  arrows  represent  mapping 
operators  which  take  a  vector  at  one  level  and  perform  a  mapping  to  the  next  lower  level. 


•  The  operators  are  generally  stochastic  and  non-linear;  therefore  the  transformations  are  not  invertible,  and 
multiple  levels  (e,g.  Level  2]  to  4])  cannot  be  properly  mapped  in  a  single  process. 

•  In  certain  V/L  modeling  tasks,  the  mathematical  mapping  operators  must  utilize  stochastic  processes  to 
yield  accurate  results.  This  is  most  notably  true  in  the  damage  operator,  where  expected-value  trans¬ 
formations  lead  to  incorrect  results. 

•  The  effect  of  target  damage,  whether  simultaneous  (e.g,  the  striking  of  two  or  more  artillery  fragments), 
time-ordered  (e.g,  a  volley  of  impinging  rounds),  or  caused  by  multiple  physical  mechanisms  {e,g.  blast 
and/or  shock  and/or  fire)  must  be  aggregated  at  Level  2],  the  damage  vector  level.  Clearly  if,  for  example, 
a  personnel  carrier  were  struck  by  two  artilleiy  fragments  which  each  happened  to  kill  the  same  compo¬ 
nents),  then  the  damage  at  Level  2]  from  both  fragments  would  be  no  greater  than  the  damage  from  either 
fragment  individually.  However  the  standard  method  for  computing  artillery  effectiveness  for  the  past  30 
years  has  been  to  compute  an  approximation  of  a  damage  vector  (Level  2]  metric)  for  each  fragment,  map 
the  resulting  damage  directly  to  a  Loss-of-Function  (Level  4]  metric),  and  then  to  combine  each  of  the  sys¬ 
tem  LoFs  (using  a  formula  resembling  the  survivor  rule  for  combining  probabilities)  to  get  an  overall  vehi¬ 
cle  LoF.  The  merging  of  individual  damage  mechanisms  at  Level  4]  rather  than  at  Level  2]  is  a  major  diffi¬ 
culty  in  a  substantial  amount  of  today’s  battlefield  simulations.  A  corollary  to  this  observation  is  that  when¬ 
ever  a  battlefield  threat  is  played  against  a  particular  target,  that  target  should  not  be  assumed  to  be  pristine, 
but  reflect  all  prior  aggregated  damage  except  that  which  may  have  been  repaired. 

•  Level  4]  metrics,  essentially  battlefield  utilities,  cannot  be  observed  through  testing.  In  addition,  to  get  to 
Level  4]  via  the  ^  mapping  operator  requires  the  incorporation  of  tactics,  doctrine,  threat  systems  and 
battlefield  environment;  this  division  of  labor  is  clearly  in  the  province  of  the  force-on-force  modeler,  not 
the  vulnerability  analyst. 

•  Each  level  of  the  taxonomy  as  it  is  applied  in  a  given  V/L  model  reflects  a  certain  level  of  granularity  (or 
resolution).  The  ability  to  perform  adequately  a  computation  at  one  level  in  the  taxonomy  requires  a  partic¬ 
ular  TniniTniim  level  of  granularity  at  the  preceding  level.  The  granularities  of  a  V/L  model  metrics  are  criti¬ 
cally  related  to  the  applicability  of  that  model  to  a  particular  analysis  purpose. 

6.  Applications  of  the  V/L  Taxonomy 

Since  the  original  notion  of  the  V/L  Taxonomy  was  generated,  many  extensions  have  been  made.  Delineations  of 
what  belong  at  the  levels  vice  what  constimte  operators,  the  role  of  the  process  structure  in  V/L  model  accuracy  and 
the  properties  of  operators  have  been  made  clear.^"^  The  V/L  Taxonomy  has  been  used  to  propose  Live-Fire  Test 
strategies^  which,  to  the  maximum  extent  known,  yield  predictive  metrics  compatible  with  field  observables  and  are 
also  amenable  to  statistical  validation  procedures.  It  has  been  applied  to  the  description  of  personnel  vulnerability 
and  die  description  of  Reliability,  Availability  and  Maintainability  (RAM).^  It  is  used  across  ARL/SLAD  for  couch¬ 
ing  metrics  and  transformation  operators  in  clear  and  unambiguous  ways,  not  just  for  ballistic  threats,  but  for  Elec¬ 
tronic  Warfare  and  Chemical  Threats  as  well. 


3.  Michael  W.  Staiks,  Assessing  the  Accuracy  of  Vulnerability  Models  by  Comparison  to  Vulnerability  Experiments,  Ballistic  Research  Labo- 
ratory  Technical  Report  BRL-TR-3018,  July  1989. 

4.  Paul  H.  Deitz,  Jill  H.  Smith  and  John  H.  Suckling,  Comparisons  of  Field  Tests  with  Simulations:  Abrams  Program  Lessons  Learned,  Pro¬ 
ceedings  of  the  XXVm  Annual  Meeting  of  the  Army  Operations  Research  Symposium,  11-12  October,  1989,  Ft.  Lee,  VA,  pp. 
108-128;  also  Ballistic  Research  Laboratory  Memorandum  Report  BRL-MR-38i4,  March  1990. 

5.  J.  Terrence  Klopcic,  Michael  W.  Starks,  James  N.  Walbert,  A  Taxonomy  for  the  VulnerabilitylLethality  Analysis  Process,  Ballistic  Research 
Laboratory  Memorandum  Report  BRL-MR-3972,  May  1992. 

6.  James  N.  Walbert,  Lisa  K.  Roach  and  Mark  D.  Burdeshaw,  Current  Directions  in  the  Vulnerability/Lethality  Process  Structure,  Army  Re¬ 
search  Labwatory  Technical  Report  ARL-TR-296,  October  1993. 

7.  Paul  H.  Deitz,  Michael  W.  Starks,  Jill  H.  Smith  and  Aivars  Ozolins,  Current  Simulation  Methods  in  Military  Systems  Vulnerability 
ment.  Proceedings  of  the  XXIX  Annual  Meeting  of  the  Army  Operations  Research  Symposium,  held  10-1 1  October  1990,  Ft  Lee,  VA; 
also  Ballistic  Research  Laboratory  Memorandum  Report  BRL-MR-3880,  November  1990. 

8.  Michael  W.  Starks,  Improved  Metrics  for  Personnel  Vulnerability  Analysis,  Ballistic  Research  Laboratory  Memorandum  Report  BRL- 
MR-3908,  May  1991. 

9.  Lisa  K.  Roach,  Fault  Tree  Analysis  and  Extensions  of  the  VlL  Process  Structure,  Army  Research  Laboratory  Technical  Report  ARL- 
TR-149,  June  1993. 

10.  William  J.  Hughes,  A  Taxonomy  for  the  Combined  Arms  Threat,  Chemical  Biological/Smoke  Modeling  &  Simulation  (M&S)  Newsletter, 
Vol.  l,No.3,FaU1995. 
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7.  V/L  Taxonomy  and  Survivability 

A  number  of  times  we  have  emphasized  the  importance  of  not  confounding  the  different  kinds  of  V/L  metrics  asso¬ 
ciated  with  Levels  2],  3]  and  4].  The  data  associated  with  a  Level  2]  metric  is  straightforward —  it  is  simply  an 
accounting  of  damaged  or  killed  components.  At  Level  3],  the  metrics  are  capability.  Capability  measures  can  be 
defined  clearly  in  terms  of  measureables  such  as  top  speed,  minimum  speed,  rate  of  acceleration,  rate  of  fire,  etc. 
Level  4]  metrics  may  best  be  thought  of  as  utilities  and  therefore  are  dimensionless.  It  may  be  helpful  to  review  a 
hypothetical  example  of  an  aircraft  performing  a  military  mission.  With  a  view  to  Fig.  1 ,  let  us  assume  that  a  missile 
attack  has  led  to  severing  a  fuel  line  to  one  of  two  engines  on  the  aircraft.  The  damage  vector  at  Level  2]  is  damage 
vector  of  one  element,  one  killed  fuel  line.  Applying  the  capability  operator  ^  to  the  damage  vector  gives  the  fol¬ 
lowing  result  at  Level  3];  the  aircraft  is  able  to  fly  straight  and  level,  but  not  clmb.  Now  we  examine  the  Military 
Utility  Operator,  To  apply  this  operator,  a  number  of  missions  must  be  defined.  In  one  mission,  it  might  be 
necessary  to  climb  rapidly  to  avoid  ground  ordnance.  In  mission  two,  it  might  only  be  necessary  to  maintain  level 
flight.  Thus  we  could  define  two  ^  mapping  operators.  In  the  case  of  the  first,  the  damaged  aircraft  could  not  ful¬ 
fill  the  mission  and  would  have  a  utihty  of  zero.  In  the  case  of  the  second  mission,  the  mission  could  be  supported, 
giving  a  utility  of  one.  One  can  envision  missions  which  when  applied  against  partially  performing  platforms  would 
result  in  partial  utility  [0.0  <  U  <  1.0].  Given  some  set  of  mission  utilities,  it  is  flien  possible  to  develop  an  expected 
utility,  averaged  over  some  set  of  missions. 

Often  utilities,  averaged  or  otherwise,  are  used  by  the  community  of  war  gamers.  The  utilities  are  often  simply 
defined  to  be  probabilities  of  a  certain  class  of  kill.  The  utility,  on  the  same  interval  as  a  probability,  is  used  in  the 
war  game  to  make  a  draw,  assuming  a  ballistic  encoimter.  Based  on  the  outcome,  the  platform  may  be  removed 
from  the  conflict  Potentially  three  errors  are  committed  by  this  practice. 

1]  Binning  a  vulnerability  metric  in  a  war  game  scenario  which  has  already  been  burned  by  a  vulnerability 
analyst  The  war  gamer  should  take  the  capability  information  from  Level  3]  and  play  that  characterization 
of  the  platform  in  his  mission  encounter.  The  war  game  will  then  define  the  utility  of  the  (damaged)  plat¬ 
form. 

2]  Averaging  two  or  more  utilities;  Often  averages  are  performed  over  the  outcomes  of  multiple  binning  pro¬ 
cesses.  This  is  legitimate  mathematically.  However  a  major  problem  occurs  when  an  average  utility  is 
applied  to  a  specific  mission.  They  may  in  fact  be  very  different  numbers  leading  to  highly  inaccurate  con¬ 
clusions. 

3]  Ihming  a  utility  into  a  probability:  A  practice  which  has  seen  widespread  use  in  the  ground  arena  has  been 
to  argue  that  an  average  battlefield  utility  is  equivalent  to  the  probability  of  total  loss  of  the  modeled  capa¬ 
bility. 

This  third  practice  has  been  shown  wanting  for  many  years,^  but  the  numbers  provided  by  the  V/L  community  to 
its  customers  are  still  referred  to  as  “probabilities  of  kill”  or  “expected  loss-of-function”,  even  if  at  their  founda¬ 
tion,  they  may  suffer  from  some  or  all  of  these  three  serious  problems. 

Finally,  if  a  series  of  utilities  is  derived  as  a  fimction  of  ballistic  threats  introduced  at  Level  1]  and  played  against  a 
number  of  missions  in  the  O  utility  mapping,  it  would  appear  that  we  have  proper  measures  for  what  might  be 
called  ballistic  survivability.  We  have  ignored  all  other  factors  in  survivability  including  the  probability  of  detection 
at  various  wavelengths,  agility,  etc.  One  can  fliink  of  overall  survivability  as  the  combined  output  of  an  O,  .  utility 
map  utilizing  not  only  the  Level  3]  capability  metrics  of  ballistic  measures,  but  using  the  similar  capability’ metrics 
from  all  of  the  other  disciplines  which  affect  survivability. 


11.  James  R.  Rapp,  An  Investigation  of  Alternative  Methods  for  Estimating  Armored  Vehicle  Vulnerability,  Ballistic  Research  Laboratory 
Memorandum  Report  ARBRL-MR-03290,  July  1983. 

12.  Michael  W.  Starics,  New  Foundations  for  Tank  Vulnerability  Analysis  (With  1991  Appendix),  Ballistic  Research  Laboratory  Memoran¬ 
dum  Report  BRL-MR-3915,  May  1991. 
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In  the  Army,  there  is  now  movement  to  repair  these  logical  lapses.  Recent  work  with  battlefield  modelers'’"^  has 
shown  that  a  war  game  is  the  singular  place  where  battlefield  utility  should  be  estimated,  and  this  in  a  robust  situa¬ 
tion  where  tactics,  doctrine  and  appropriate  stochasticism  can  be  properly  played.  Thus  the  Level  3]  to  4]  mapping 
for  vulnerability/lethality  must  be  played  along  with  all  of  the  other  relevant  platform  metrics  outside  of  the  pure 
V/L  milieu. 


O' 


Appendix  A 

Definitions  of  Lethality,  Vulnerability  and  Survivability^ 


VulnerabUity 

The  characteristics  of  a  system  that  cause  it  to  suffer  a  degradation  [loss  or  reduction  of  capability  to  perform  the 
designated  mission(s)]  as  a  result  of  having  been  subjected  to  a  hostile  environment  on  the  battlefield.  It  is  generally 
an  assumption  in  vulnerability  studies  that  the  threat  warhead  has  engaged  the  target. 

Lethality 

The  ability  of  a  system  to  cause  the  loss  of,  or  a  degradation  in,  the  ability  of  a  target  system  to  complete  its  desig¬ 
nated  mission(s).  Often  for  direct-fire  weapons,  the  delivery  of  the  warhead  firom  launch  to  target  impact  is  integral 
to  the  lethality  analysis.  For  indirect-fire  weapons,  studies  often  begin  with  warhead  initiation  in  the  neighborhood 
of  the  target. 

Survivability 

The  capability  of  a  system  (resulting  from  the  synergism  among  personnel,  materiel,  design,  tactics  and  doctrine)  to 
avoid,  withstand  or  recover  in  hostile  (man-made  and  natural)  environments  without  suffering  an  abortive  impair¬ 
ment  of  its  ability  to  accomplish  its  designated  mission.  If  the  two  facets  under  control  of  a  weapons  designer, 
materiel  and  design,  are  lumped  into  System  Characteristics,  and  the  effects  of  persormel  are  distributed  appropri¬ 
ately  over  the  three  remaining  variables  of  Characteristics,  Tactics  and  Doctrine,  then  Survivability  can  be  written 
functionally  as: 


Survivability  =  f  [Threat  (Characteristics,  Tactics,  Doctrine), 

Battlefield  Environment, 

System  (Characteristics,  Tactics,  Doctrine)] 


t  Private  communication  with  personnel  at  TRACAVSMR.  War  Gamers  at  TRAC  are  modifying  a  version  of  CASTFOREM  so  as  to  ac¬ 
commodate  ballistic  inputs  at  Levels  2]  and  3].  These  war  games,  as  a  consequence  of  their  outputs,  provide  a  Level  3]  to  Level  4]  map¬ 
ping  (i.e.  utility  weighting). 

t  Private  communication  with  W.  J.  Brooks,  Jr.,  of  AMSAA.  A  DMSO-funded  ATTD  Program  directed  by  Mr.  Brooks  is  being  configured 
so  as  to  accept  Level  2]  (disabled  components)  and  Level  3]  (degraded  capability)  metrics  from  V/L  models. 

□  Consistent  with  Defense  Acquisition  Management  Policies  and  Procedures,  DoD  Instruction  5000.2,  February,  1991. 
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MODELING  BALLISTIC  LIVE-FIRE  EVENTS 


Dr.  Paul  H,  Deitz  and  Mr  Richard  Saucier 
U.S.  Army  Research  Laboratory,  Aberdeen  Proving  Ground,  MD  21005-5068 

ABSTRACT 

This  paper  explains  the  need  for  a  stochastic  vulnerability  model  to  support  the  analysis  of 
live-lire  testing.  The  history  of  the  development  and  use  of  such  a  model  over  the  last  decade  is 
summarized  by  demonstrating  the  need  for  new  methodologies  and  die  establishment  of  the 
SQuASH  vulnerability  model.  A  brief  review  is  made  of  the  various  assessment  efforts  made  to 
compare  SQuASH  model  outputs  with  various  Abrams  Live-Fire  Test  results.  This  has  lead  to  a 
model  improvement  plan  for  upgrading  SQuASH.  The  incorporation  of  the  upgraded  model  into 
the  MUVES  suite  of  vulnerability  codes  and  its  application  to  the  upcoming  Armored  Gun  Sys¬ 
tem  (AGS)  Live-Fire  program  are  described. 


INTRODUCTION 

Nearly  a  decade  ago,  the  U.S.  began  a  new  form  of  vulnerability  experimentation  called  Live-Fire  Testing  (LFT) 
(ref.  1).  In  LFT,  a  complete  vehicle,  such  as  a  tank  or  armored  personnel  carrier,  is  placed  in  full  battle  readiness, 
engine  running,  fiiU  load  of  fuel  and  ammunition,  and  fired  at  with  an  overmatching  threat.  Only  the  absence  of  a 
live  crew  compromises  actual  encounter  realism.  Congressional  legislation  had  been  passed  recognizing  that  in  spite 
of  design  limits  defining  absolute  protection,  systems  should  nevertheless  be  tested  according  to  threats  expected  to 
be  encountered.  Many  such  threats  could  be  overmatching.  The  issue  was  to  mitigate  and  ameliorate  such  events. 
In  addition,  LFT  can  uncover  vulnerabilities  not  foreseen  by  vehicle  designers  and  improve  survivability. 

The  first  LFTs  occurred  against  the  Ml  13  armored  personnel  carrier.^  For  the  most  part,  these  results  were  non- 
controversial.  By  1985,  testing  had  begim  on  the  more  modem  Bradley  fighting  vehicle.  To  accompany  field  test¬ 
ing,  the  program  test  plans  required  that  vulnerability  models  be  used  both  to  predict  and,  subsequently,  to  be 
upgraded  by  actual  LFT  results.  As  the  test  proceeded  and  the  results  were  compared  to  model  predictions,  an 
apparent  pattern  of  disagreement  began  to  emerge.  Critics  in  the  Office  of  the  Secretary  of  Defense  (OSD)  ques¬ 
tioned  the  fidelity  of  existing  ballistic  vulnerability  modeling.  As  the  Vulnerability/Lethality  Division  (VLD)  of  the 
former  U.S.  Army  Ballistic  Research  Laboratory  (BRL)^  headed  into  the  Abrams  Live-Fire  program,  a  goal  was  set 
to  develop  a  new  model  —  one  designed  to  simulate  actual  live-fire  events,  including  the  statistical  variations  com¬ 
monly  encountered. 

DEVELOPMENT  OF  THE  SQuASH  MODEL 

The  SQuASH  model  was  developed  specifically  for  the  purpose  of  providing  a  tool  for  predicting  and  under¬ 
standing  live-fire  events.  No  existing  model  was  adequate  for  this  purpose;  either  they  did  not  predict  outcomes  that 
could  actually  be  measured  in  the  field  or  they  did  not  account  for  the  variability  of  live-fire  outcomes  —  or  they 
were  deficient  in  both  of  these  areas. 

Model  Requirements  to  Address  Live-Fire  Testing 

When  the  need  for  vulnerability  modeling  to  support  LFT  arose  in  1985,  a  number  of  insights  began  to  emerge: 
t  Bradley  live-fibre  actuaUy  began  before  the  Ml  1 3  tests,  but  the  Ml  1 3  firings  were  completed  first. 

t  On  30  September  1992,  the  U.S.  Army  Ballistic  Research  Laboratory  (BRL)  was  deactivated  and  subsequently  became  part  of  the  U.S. 

Army  Research  Laboratory  (ARL)  on  1  Ocotober  1992. 
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observable  damage  or  vehicle  capability.  Vulnerability  models  have  historically  done  a  poor  job  of  devel¬ 
oping  metrics  that  are  observable  in  the  field.  The  emphasis  in  vulnerability  modeling  today  is  on: 

—  Direct  battle  damage  (i.e.,  “killed”  or  nonfunctioning  components)  and 

—  Platform  capability  (the  relationship  between  “killed”  components  and  measurable  platform  function, 
rate  of  gun  fire,  top  speed,  etc.). 

Battlefield  utility,  which  is  not  an  observable,  is  now  seen  to  be  the  proper  province  of  the  force-on-force 
modeler. 

•  Few  vulnerability  models  reflected  the  variability  that  is  intrinsic  to  many  ballistic  interactions.  When  pen- 
etrators  strike  a  target  and  perforate  the  skin,  the  armor  spalls  into  numerous  fragments  of  varying  mass, 
velocity,  shape,  and  orientation.  However,  the  existing  models  all  converged  on  a  single,  expected-value  for 
the  final  result,  rather  than  yielding  statistical  distributions  of  possible  outcomes.  Some  models  attempted 
to  calculate  the  probability  of  damaging  components  individually  along  a  particular  shotline  but  said  noth¬ 
ing  about  the  probability  of  damaging  components  in  combination  with  one  another. 

In  addition  to  establishing  the  need  for  a  new  model,  this  retrospection  of  existing  vulnerability  models  also 
made  it  clear  that: 

•  There  is  a  dearth  of  information  concerning  many  of  the  mechanisms  of  physical  damage  from  ballistic 
interactions. 

•  Tliere  are  many  ballistic  mechanisms,  which  can  cause  significant  damage,  that  were  not  modeled  at  all. 
TTiese  typically  included  blast,  shock,  fire,  and  toxic  fumes. 

Another  important  by-product  of  this  reexamination  has  been  the  establishment  of  a  formal  framework  within 
which  to  understand  and  categorize  the  elements  and  properties  of  vulnerability/lethality  (V/L).  Termed  the  V7L 
Taxonomy  (ref.  2),  it  provides  a  method  to  decompose  the  elements  of  V/L  into  a  sequence  of  simpler  constituent 
parts  called  leveb.  The  levels  relate  to  each  other  in  a  specific  processing  order  and  are  fundamentally  different. 
Each  has  its  unique  and  appropriate  use  in  the  general  scheme  of  V/L  assessment  and  are  related  to  each  other  by 
mapping  operators.  It  can  be  seen  that  most  (if  not  all)  of  these  mapping  operators  must  be  stochastic  in  order  to 
support  the  variability  so  intrinsic  to  many  of  the  V/L  processes.  A  further  characterizing  property  of  the  vulnerabil¬ 
ity  levels  is  the  degree  to  which  specific  metrics  are  aggregated  (i.e.,  lumped  together)  versus  refined  (i.e.,  subdi¬ 
vided  into  smaller  elements).  The  word  granularity  is  used  to  describe  this  property.  Since  the  earliest  notions  of 
the  V/L  Taxonomy  were  first  expressed  (ref.  2),  much  progress  has  been  made  to  formalize  and  extend  this  concept. 
TTie  concept  of  the  V/L  Taxonomy  has  applicability  ranging  from  how  platform  component  damage  relates  to  plat¬ 
form  performance  to  how  platform  metrics  should  feed  force-on-force  models.  (The  reader  is  referred  to  ref.  2  for  a 
current  summary  of  this  framework  and  its  ramifications.) 

Need  for  a  Stochastic  Model 

To  address  the  shortcomings  described  previously,  in  1985  the  V/L  Division  (VLD)  of  BRL  embarked  on  a  sig¬ 
nificant  initiative  to  establish  a  new  kind  of  vulnerability  code,  which  would  explicitly  embody  observable  model 
metrics  supported  by  stochastic  methods.  The  new  code,  called  SQuASH  (Stochastic  j2«antitative  Analysis  of  Sys¬ 
tem  ffierarchies),  was  applied  to  some  48  Abrams  shots  performed  in  the  1986  time  frame.  SQuASH  supported  the 
following  four  sources  of  variability,  subject  to  random  sampling: 

•  Weapon  Hit  Point  —  sampled  in  the  neighborhood  of  the  intended  impact  point. 

•  Warhead  Performance  —  from  a  Gaussian  distribution  with  mean  and  standard  deviation  obtained  from 
experimental  data. 

•  Residual  Penetrator  Deflection  —  from  a  Gaussian  distribution  with  mean  and  standard  deviation  obtained 
from  the  variability  of  experimental  data,  width  of  the  spall  cone,  and  a  Poisson  distribution  that  controls 
the  number  of  spall  fragments.  The  parameters  for  these  distributions  were  derived  from  experimental  data. 

•  Component  Pk  Characterization  — ■  from  a  Poisson  distribution  to  determine  the  number  of  impacts  on  a 
component  and  a  uniform  distribution  to  perform  a  Bernoulli  trial  on  whether  the  component  was  “killed” 
or  “not-killed.” 

Because  of  the  paucity  of  knowledge  for  many  sources  of  damage,  as  well  as  insufficient  time  to  code  new  algo¬ 
rithms,  the  only  damage  mechanisms  modeled  for  this  analysis  were  main  penetrator  (including  residual  penetrator) 
and  behind-armor  debris  spall. 


14 


and  behind-annor  debris  spall. 

ASSESSMENT  OF  SQu ASH  —  LESSONS  LEARNED 

Following  the  Abrams  program,  a  number  of  significant  efforts  were  made  to  compare  the  model  results  with 
the  field  observables.  This  effort  proved  daunting  as  the  SQuASH  model  exercise  indicated  the  possibility  of  more 
than  10^  individual  combinations  of  component  “kill”  records  for  particular  warhead-target  encounters.  A  thou¬ 
sand,  or  even  ten  thousand,  Monte  Carlo  replications  with  the  model  may  not  be  sufficient  to  produce  the  exact 
sequence  of  damaged  components  observed  in  the  field.  Validation  of  a  stochastic  vulnerability  model  was  —  and 
still  remains  —  problematical. 

Early  Assessment  of  SQuASH 

The  initial  published  results  addressing  model  validation  for  Abrams  (refs.  3-5)  live-fire  shots  resulted  in  the 
following  conclusions  (taken  from  ref.  5): 

•  Tests  on  the  probability  of  armor  perforation  were  in  good  agreement  with  model  predictions. 

•  Tests  on  the  probability  of  catastrophic  “kill”  were  also  in  good  agreement  with  model  predictions. 

•  Tests  on  the  Mobility  “Kill”  prediction  were  at  the  85%  agreement  level  with  model  predictions. 

•  Tests  on  the  Fire  Power  “Kill”  prediction  were  at  the  33%  level  of  agreement  with  model  predictions. 

A  number  of  other  issues  came  to  the  fore.  It  was  clear  that  fire,  secondary  spall,  ricochet,  and  blast  sometimes 
played  dominant  roles.  The  absence  of  algorithms  in  the  model  to  describe  these  potential  sources  of  damage 
needed  to  be  redressed.  In  addition,  unless  there  were  major  disagreements  between  the  model  and  the  field  tests,  the 
very  complexity  of  ballistic  interactions  —  with  the  concomitant  space  of  greater  than  10®  discrete  outcomes  to  be 
compared  with  a  small  number  of  single  test  results  —  made  statistical  inference  problematic. 

In  the  time  since  1986,  various  upgrades  to  SCJuASH  have  been  made  and  the  model  has  been  used  in  a  number 
of  Army  Live-Fire  programs.  These  include  MlAl,  Paladin,  M1A2,  and  T-72.  To  avoid  confusion,  note  that  the  ini¬ 
tial  configuration  of  SQuASH  used  for  the  Abrams  Ml  A1  live-fire  program  was  written  in  FORTRAN.  The  version 
of  SQuASH  that  is  being  incorporated  into  MUVES  (described  in  subsequent  sections)  is  written  in  the  C  language. 

Independent  Assessment  of  SQuASH 

In  addition  to  the  previous  in-house  comparison,  VLD  also  contracted  with  The  SURVICE  Engineering  Com¬ 
pany  in  1991  to  perform  an  independent  assessment  of  SQuASH.  Their  findings  will  be  discussed  shortly. 

Meanwhile,  in  1993  the  Army  was  involved  in  the  Abrams  Ml  A2  upgrade.  To  decrease  the  cost  of  testing,  the 
Department  of  the  Army  (DA)  proposed  to  substitute  SQuASH  predictions  for  a  portion  of  LFT.  To  estimate  the 
risk  associated  with  this  strategy,  OSD  tasked  the  Institute  for  Defense  Analyses  (IDA)  to  perform  an  independent 
assessment  of  SQuASH.  In  a  set  of  charts,^  IDA  appears  to  have  utilized  a  novel  strategy  in  which  the  probabilities 
of  damaging  particular  components  using  multiple  threats  at  various  hit  locations  are  aggregated  over  many  tests. 
Via  this  strategy,  the  paucity  of  field  tests  matching  each  model  exercise  would  appear  to  have  been  somewhat  miti¬ 
gated.  Based  on  these  findings,  IDA  drew  the  following  three  conclusions: 

•  SQuASH  is  a  poor  predictor  of  damage  at  the  component  level. 

•  The  number  of  components  where  the  damage  was  unexpected  (Pj^  <  .  05)  is  comparable  to  or  exceeds  the 
number  where  the  damage  was  expected. 

•  A  large  fraction  of  the  damaged  components  was  never  reported  by  the  model. 

Of  the  259  “killed”  components  used  in  the  IDA  study,  SQuASH  correctly  predicted  damage  to  49%  (127)  and 
incorrectly  predicted  no  damage  to  51%  (132).  The  latter  category  can  be  further  subdivided  into  two  groups: 

1 .  Component  was  hit  but  not  predicted  to  be  damaged  —  accounting  for  21  %  (54  components)  and 

2.  Component  was  not  hit  by  a  ray  —  accounting  for  30%  (78  components). 

IDA  did  not  provide  insight  into  the  possible  origins  of  SQuASH- Abrams  LFT  discrepancies.  From  our  analy¬ 
ses,  these  two  categories  correspond  to  separate  issues  and  are  being  addressed  by  different  strategies.  The  first  cate¬ 
gory  illustrates  a  deficiency  of  modeling  the  vulnerability  of  critical  components.  Past  modeling  used  an  average 

t  To  date,  no  fonnal  report  of  the  findings  has  been  published  by  IDA. 
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probability  of  kill  (Pjt),  averaged  over  hit  location  and  direction  of  penetration.  Due  to  the  inherent  variability  of 
component  vulnerability,  a  distribution  of  possible  outcomes  rather  than  a  single  expected  value  may  be  more  appro¬ 
priate  —  and  we  are  exploring  this  option.  However,  another  factor  that  may  contribute  to  the  discrepancy  —  and 
may  even  be  more  important  —  is  the  use  of  a  single  draw  on  the  component  to  produce  a  binary  kill/no-kill  out¬ 
come.  Even  if  the  component  is  modeled  extremely  well,  the  expected  number  of  Bernoulli  trials  required  to  realize 
a  “kill”  is  1/Pjt,  where  Pj^  is  the  component’s  probability  of  “kill.”  Thus  an  unintended  result  of  the  Bernoulli  trial 
approach  is  to  effectively  under-sample  the  components  having  low  Pjt  values.  We  now  believe  that  a  better 
approach  may  be  to  use  the  actual  Pj^  value  in  the  Criticality  Analysis  (see  ref.  6,  for  an  example  of  the  latter)  — 
and  the  methodology  is  being  changed  to  accomplish  this. 

The  second  category  illustrates  a  geometric  sampling  problem:  If  SQuASH  never  samples  a  component,  it  can 
never  be  regarded  as  damaged.  The  methodology  being  developed  in  the  new  SQuASH  model  will  be  using  a  differ¬ 
ent  approach  altogether  for  generating  spall  fragments  (see  Improvements  to  SQuASH  below),  as  well  as  modem 
recursive  techniques  to  produce  secondary  burst  points. 

In  general,  the  conclusions  of  the  SURVICE  study  (ref.  7)  were  not  substantially  different  from  the  IDA  study, 
although  they  do  contain  more  detail.  However,  the  SURVICE  study  used  a  more  stringent  statistical  test  called  the 
Modified  Ordering  of  Probabilities  Test  (ref.  8). 

MUVES  —  NEW  CONTEXT  FOR  SQuASH 

In  1986,  concurrent  with  the  development  of  SCJuASH,  the  VLD  initiated  a  project  to  develop  a  computer  archi¬ 
tecture,  written  in  the  C  language,  that  would  be  modular  in  nature,  strongly  coupled  to  the  UNIX  operating  system, 
and  would  minimize  vulnerability  code  redimdancy.  This  effort  resulted  in  a  computer  environment  called  MUVES 
(Modular  C/NTX-based  Vulnerability  Estimation  Suite)  (ref.  9).  The  development  was  made  possible  by  a  number  of 
new  insights,  including: 

•  The  actual  vulnerability  process  can  be  broken  into  specific  building  elements.  These  elements  can  calcu¬ 
late  possible  damage  in  uniform  ways  regardless  of  the  threat-material  class,  can  aggregate  damage  in  con¬ 
sistent  ways  to  estimate  component  “kill,”  and  can  map  component  “kill”  to  platform  capability  in  the 
same  general  fashion. 

•  Each  vulnerability  code  is  composed  principally  of  support  modules  that  are  not  threat-target  specific. 
Eighty-five  percent  of  the  code  includes  modules,  for  example,  to  manage  memory,  interrogate  geometry, 
interpolate  tables,  and  draw  random  numbers.  All  of  these  modules  should  be  applicable  to  many  classes  of 
vulnerability  computation  for  many  threats  and  many  targets. 

•  Hie  computer  methods  and  techniques  historically  used  to  code  vulnerability  models  are  inadequate  to  the 
task  of  rapid  upgrade,  code  W&A,  extensibility  to  new  encounter  conditions,  and  ability  to  share  code 
among  various  threat-target  classes. 

Over  the  past  six  years,  the  Ballistic  Vulnerability/Lethality  Division  (BVLD)  has  been  moving  its  vulnerability 
codes  under  this  single,  consistent  MUVES  environment.  The  first  code  integrated  into  MUVES  was  the  direct-fire, 
lumped-parameter.  Compartment  Model,  historically  called  VAMP  (ref.  10).  The  resulting  code  has  supported  many 
direct-fire  studies  over  the  past  years.  In  the  summer  of  1995,  an  aircraft-missile  model  called  MAVEN  (Modular 
Air-system  Vulnerability  Estimation  A^etwork)  (ref.  11)  was  placed  in  production.  The  first  operational  capability 
embodied  armor-piercing  (AP)  threats  against  aircraft  and  was  used  for  live-fire  predictions  in  the  current  Apache- 
Longbow  program  (ref.  12).  Now  in  final  beta  testing  is  an  indirect-fire  model  called  SAFE  (Statistical  Analysis  of 
Fragment  Effects)  (ref.  13).  A  major  improvement  for  indirect-fire  weapons,  including  artillery  munitions,  SAFE 
can  assess  multiple  burst  points,  proper  target  perspective,  random  fragments  (including  mass,  velocity,  shape,  and 
orientation),  and  aggregate  damage  correctly  at  the  vehicle  component  level.  SCJuASH  is  also  moving  to  the 
MUVES  environment.  Not  only  is  the  overhead  in  dealing  with  the  SQuASH  FORTRAN  environment  too  high,  it  is 
incapable  of  supporting  the  proper  inclusion  of  new  damage  mechanisms. 

SQuASH  IMPROVEMENTS  AND  APPLICATIONS 

Methodology  Improvements 

Wth  the  intent  of  redressing  the  modeling  deficiencies  that  were  identified  previously  (see  ASSESSMENT  OF 
SQuASH — LESSONS  LEARNED)^  the  following  changes  to  the  methodology  are  being  implemented: 
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•  Upgraded  Kinetic  Energy  (KE)  Long  Rod  Armor  Penetration:  Based  on  the  research  of  Alekseevskii 
(ref.  14)  and  Tate  (ref.  15),  this  methodology  applies  to  a  wide  variety  of  threat-target  interactions  by 
accounting  for  hydrodynamic  flow  of  both  target  and  penetrator  that  can  arise  in  the  hypervelocity  regime. 
This  methodology  also  makes  explicit  use  of  physical  quantities,  such  as  Brinell  hardness,  that  can  be  var¬ 
ied  in  a  stochastic  maimer  when  it  is  appropriate  to  do  so  —  such  as  when  the  penetrator  is  close  to  the  bal¬ 
listic  limit  V50  of  perforation. 

•  New  Spall  Characterization:  A  substantial  effort  has  been  expended  to  improve  (ref.  16)  the  SQuASH 
spall  model.  The  upper  section  of  Fig.  1  shows  a  flash  radiograph  of  a  shaped-charge  jet  following  armor 
penetration.  An  elliptical  debris  cloud  is  evident.  The  lower  section  shows  the  geometric  characterization 
of  the  shell  shape  and  velocity  field  used  to  model  this  phenomenon. 


Figure  1.  Behind-Armor  Debris  Characterization 

The  image  on  die  left  in  Figme  2  shows  a  three-dimensional  rendering  of  a  spall  cloud  compute  with 
expected-value  parameters  for  each  surface  element  on  the  cloud  topology.  The  image  on  the  right  in  this 
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figure  shows  the  same  plot  with  stochastic  spall  generation  enabled. 


Figure  2.  Three-Dimensional  Renderings  of  Spall  Cloud  (Images  provided  by  Gary  Moss) 

•  Multiple  Barrier  Penetration:  The  code  will  propagate  the  threat  (spall  or  penetrator)  until  it  comes 
to  rest  or  exits  the  vehicle.  The  FORTRAN  version  of  SQuASH  did  not  track  spall  beyond  the  first 
barrier,  and  broken  penetrator  pieces  were  not  tracked  beyond  the  sixth  barrier. 

•  Accounting  for  Ricochet:  Ricochet  was  not  accounted  for  in  flie  FORTRAN  version  of  SQuASH, 
although  the  LFT  assessors  noted  the  occurrence  of  ricochet  in  the  live-fire  tests.  SQuASH  will  be 
using  a  ricochet  criterion  published  by  Tate  (ref.  17)  that  is  based  upon  penetrator  velocity  and  target 
obliquity.  When  the  ricochet  criterion  is  satisfied,  the  code  will  go  on  to  compute  the  ricochet  angle 
and  residual  velocity  (ref.  18). 

•  Upgraded  Personnel  Incapacitation:  The  FORTRAN  version  of  SQuASH  used  the  Kokinakis-Sper- 
razza  criteria  for  crew  incapacitation  (ref.  19).  The  MUVES  version  of  SQuASH  will  use  “Ballistic 
Dose”  (ref.  20).  This  is  a  combination  of  mass,  velocity,  and  number  of  fragment  hits  that  was  formu¬ 
lated  from  extensive  runs  of  the  ComputerMan  model  (ref.  21).  It  removes  a  number  of  limitations  of 
the  older  methodology. 

•  New  Component  Characterization:  The  FORTRAN  version  of  SQuASH  performed  a  Bernoulli  trial 
on  each  component  that  was  hit  to  get  a  binary  “kill/no-kill”  outcome.  Since  the  F*  of  the  component 
will  be  known  as  a  function  of  the  encounter  conditions,  flie  MUVES  version  of  SQuASH  will  output 
the  F*  value  itself.  This  value  will  then  be  passed  along  from  each  component  and  combined  using 
fault-tree  algebra.  This  change  in  methodology  should  go  a  long  way  toward  alleviating  the  sampling 
problem  that  has  been  identified  in  the  EDA  and  SURVICE  studies. 

•  New  Sampling  Procedures:  The  FORTRAN  version  of  SQuASH  used  the  expected  number  of  frag¬ 
ments  along  a  trajectory,  whereas  die  MUVES  version  of  SQuASH  brings  more  fidelity  to  the  process 
by  specifically  generating  a  ray  for  each  fragment.  This  allows  for  more  realism  and  a  more  natural 
application  of  stochasticism.  The  MUVES  version  of  SQuASH  will  have  a  library  of  20  continuous 
distributions  and  7  discrete  distributions  to  simulate  various  random  processes.  Furthermore,  this 
library  is  extensible  so  that  the  user  can  use  an  empirical  distribution  or  a  user-specified  function  to 
represent  a  given  stochastic  process.  Among  the  stochastic  processes  that  the  MUVES  version  of 
SQuASH  will  consider  as  the  model  is  fine-tuned  are: 
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—  Hit  Location 

—  Penetration  Depth  (based  upon  variation  of  material  properties) 

—  Ballistic  Limit  Velocity  (based  upon  spread  about  the  V50  value) 

—  Penetrator  Breakup  (based  on  velocity,  obliquity,  and  yaw) 

—  Deflection  of  KE  Long  Rod  Penetrators 
—  Ricochet  Angle  of  KE  Long  Rod  Penetrators 
—  Spall  (fragment  velocity,  size,  direction,  and  orientation) 

—  Component 

We  note  again  that  neither  the  SURVICE  nor  IDA  study  was  able  to  determine  specific  causes  for  disagree¬ 
ment  between  SQuASH  and  the  Abrams  LFT  program.  Our  diagnosis  of  the  disagreement  is  based  upon  past 
experience,  knowledge  of  the  modeling  methodology,  and  insight.  As  a  result,  we  believe  that  the  planned 
improvements  outlined  in  this  section  are  plausible  cures  for  these  deficiencies  but,  of  course,  it  will  be  the 
actual  application  of  the  model  to  Live-Fire  that  will  determine  our  degree  of  success. 

Application  to  the  Armored  Gun  System  (AGS) 

The  first  application  for  the  MUVES  version  of  SQuASH  comes  soon  with  the  AGS  program.  Figure  3 
shows  a  shaded  rendering  of  the  BVLD-generated  image.  This  image  was  generated  with  the  BRL-CAD™ 
suite  of  supporting  utilities. 


Figure  3.  A  shaded  rendering  of  the  Armored  Gun  System  (AGS)  (Image  provided  by  Ted  Muehl) 

AGS  will  constitute  the  Army’s  new  combat  vehicle,  but  in  the  form  of  a  highly  deployable,  light-weight 
vehicle,  with  high  fire-power  and  reconfigurable  armor  protection.  Analysts  are  assembling  the  various  required 
inputs  including  target  description,  fault-trees,  penetration-fragment  parameters  and  component  functions. 
This  preparation  phase  is  particularly  challenging  due  to  1)  the  multiple  armors  being  used  on  the  AGS  and  2) 
the  paucity  of  relevant  ballistic  databases,  at  least  in  comparison  to  that  known  at  a  comparable  time  in  the 
Abrams  program. 

The  MUVES  environment  is  being  upgraded  with  the  software  improvements  previously  discussed  (see 
IMPROVEMENTS  TO  SQuASH).  The  application  of  the  new  MUVES  version  of  SQuASH  to  the  AGS  will  pro¬ 
vide  valuable  information  on  the  improvement  strategy  that  we  have  outlined  in  fliis  report. 
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Future  Improvements 

In  addition,  studies  continue  elsewhere  in  ARL  to  bring  insight  into  the  balhstic  phenomena  of  shock-blast  load¬ 
ing  and  resulting  component  damage.  As  these  complex  damage  mechanisms  are  gradually  understood  and  mod¬ 
eled,  BVLD  will  integrate  appropriate  algorithms  into  the  MUVES  environment.  This  strategy  has  already  resulted 
in  great  leveraging.  First,  mechanisms  are  easier  to  model  and  integrate  into  MUVES  because  the  support  structure 
already  exists.  Second,  when  a  new  mechanism  is  included  one  place  in  the  MUVES  environment,  it  is  available  for 
all  other  approximation  methods  sharing  this  environment.  As  the  MUVES  environment  matures  and  these  methods 
gain  sophistication,  we  expect  a  gradual  shift  away  from  the  long-existent  problems  of  code  implementation  and 
towards  the  fidelity  and  calibration  of  specific  ballistic  phenomena  and  related  platform  dysfunction. 

NEED  FOR  STATISTICS 

A  recurring  theme  throughout  the  application  of  SQuASH  to  LFT  has  been  the  need  for  statistical  measures  of 
comparison  (see  refs.  8, 22-24).  Even  the  question  of  what  test  to  use  is  not  at  all  obvious.  Different  analysts  apply 
different  tests  to  judge  the  comparison  between  model  predictions  and  field  test  results.  Furthermore,  there  are  vari¬ 
ous  levels  at  which  the  comparison  can  be  performed  —  ranging  from  component  damage  states  to  platform  perfor¬ 
mance.  Nor  is  this  problem  likely  to  be  solved  any  time  soon;  it  is  an  active,  ongoing  area  of  research. '  Even  if  we 
did  not  have  the  SQuASH  model,  we  would  still  be  faced  with  the  problem  of  drawing  statistically  significant  con¬ 
clusions  from  a  small  number  of  live-fire  tests,  each  of  which  is  one  realization  from  a  different  underlying  distribu¬ 
tion  of  possible  outcomes.  Of  course  the  problem  of  inferring  statistically  significant  conclusions  from  model  pre¬ 
dictions  vis-a-vis  live-fire  tests  extends  beyond  SQuASH  to  vurtually  all  of  the  high-resolution  V/L  models  whether 
within  or  without  the  MUVES  environment. 

CONCLUSIONS 

In  this  paper,  we  have  reviewed  a  decade  of  Live-Fire  V/L  testing  and  modeling.  The  test  programs  have 
spurred  substantial  improvements  to  the  extant  suite  of  V/L  codes,  one  of  which  is  the  SQuASH  model.  Compar¬ 
isons  of  SCJuASH  model  predictions  with  field  tests  indicate  that  fragment  damage  is  underpredicted;  other  damage 
mechanisms  must  be  added.  A  detailed  strategy  for  code  upgrade  has  been  outlined  including  not  just  specific  com¬ 
putational  fixes,  but  the  use  of  a  general  V/L  computing  environment  called  MUVES.  Currently  SQuASH  is  being 
upgraded  in  preparation  for  the  AGS  live-fire  program.  The  next  half  year  will  provide  a  set  of  new  opportunities  to 
gauge  progress  in  V/L  model  improvement 
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This  paper  presents  our  plan  for  Verification,  Validation,  and  Accreditation  (W&A)  of 
the  Stochastic  Quantitative  Analysis  of  System  Hierarchies  (SQuASH)  Model.  This  model 
was  developed  for  the  express  purpose  of  providing  a  tool  to  the  vulnerability/lethality 
(V/L)  analyst  to  use  for  planning,  analyzing,  and  assessing  Live-Fire  Test  shots  of  Armored 
Fighting  Vehicles  (AFVs).  From  its  initial  conception,  it  was  designed  to  use  a  high-resolu¬ 
tion  target  description  of  an  AFV  and  to  be  stochastic  in  nature  —  two  essential  character¬ 
istics,  we  would  argue,  to  be  able  to  meet  its  intended  purpose.  Modeling  and  simulation 
(M&S)  are  veiy  valuable  tools  for  understanding  V/L  issues  in  a  well-balanced,  cost-effec¬ 
tive  program  of  analysis  and  testing.  However,  before  SQuASH  can  be  treated  as  a  credible 
supplement  to  Live-Fire  Testing  (LFT),  it  must  first  undergo  a  comprehensive  verification 
and  validation  process.  This  paper  presents  our  plan  to  accomplish  this  goal. _ 


INTRODUCTION 

Over  a  decade  ago,  Congress  mandated  [Live  Fire  Testing,  1987]  that  new  Armored  Fighting  Vehicles  (AFVs) 
must  undergo  Live-Fire  Testing  (LFT)  before  they  are  fielded.  The  AFV  is  placed  in  full  battle  readiness,  with 
engine  running  and  a  full  load  of  fuel  and  live  ammunition,  and  subjected  to  firings  from  an  overmatching  threat 
munition.  Only  the  use  of  manikins  for  a  live  crew  compromises  the  realism  of  an  actual  battlefield  encounter. 

As  we  took  inventory  of  the  state  of  vulnerability  models  in  1985,  it  became  clear  that  existing  models  were  not 
adequate  for  predicting  live-fire  events: 

•  Few  vulnerability  models  reflected  the  variability  that  is  intrinsic  to  many  ballistic  interactions.  When 
penetrators  strike  a  target  and  perforate  the  skin,  the  armor  spalls  into  numerous  fragments  of  varying 
shape,  mass,  velocity,  and  orientation.  However,  the  existing  models  all  converged  on  a  single, 
expected-value  for  the  final  result,  rather  than  yield  statistical  distributions  of  possible  outcomes.  Some 
models  attempted  to  calculate  the  probability  of  damaging  components  individually  along  a  particular 
shotline  but  said  nothing  about  the  probability  of  damaging  components  in  combination  with  one 
another. 

•  Many  of  the  metrics  commonly  output  by  the  standard  vulnerability  models,  such  as  battlefield  utility, 
were  not  observable  by  the  vehicle  assessors.  In  fact,  none  of  the  extant  vulnerability  models  computed 
directly  observable  damage  or  platform  capability.  Vulnerability  models  have  historically  done  a  poor 
job  of  developing  metrics  that  can  be  observed  in  the  field.  The  spotlight  in  vulnerability  modeling 
today  focuses  on: 

—  direct  battle  damage  (i.e.,  “killed”  or  nonfunctioning  components)  and 

—  platform  capability  (the  relationship  between  “killed”  components  and  measurable  platform  func¬ 
tion,  such  as  rate-of-gun  fire,  top  speed,  etc.). 

Battlefield  utility,  which  is  not  an  observable,  is  now  seen  to  be  the  proper  province  of  the  force-on- 
force  modeler. 

•  There  is  a  dearth  of  information  concerning  many  of  the  mechanisms  of  physical  damage  from  ballistic 
interactions. 

•  There  are  many  ballistic  mechanisms  that  can  cause  significant  damage  but  are  not  modeled  at  all. 
These  typically  included  blast,  shock,  fire  and  toxic  fumes. 

Thus  in  1985,  SQuASH  (for  Stochastic  Quantitative  Analysis  of  System  Hierarchies)  Peitz  &  Ozolins  1988]  was 
created  to  fill  this  void  in  V/L  methodology.  Three  principles  were  adhered  to  in  the  creation  of  this  model: 
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•  The  target  description  must  be  high-resolution  so  that  components  such  as  wires  and  hydraulic  lines  are 
modeled  explicitly. 

•  The  methodology  must  be  stochastic  in  nature  to  address  the  fact  that  many  V/L  mechanisms  are  highly 
sensitive  to  the  initial  conditions  and  respond  in  a  nonlinear  fashion. 

•  The  model  must  output  observables,  such  as  damage  to  components  or  engineering  capability,  not 
simply  an  expected  vehicle  loss-of-function. 

Thus,  the  SQuASH  model  is  a  stochastic,  high-resolution,  one-on-one,  threat-target  computer  simulation  that 
uses  detailed  computer-aided  design  (CAD)  target  descriptions  to  assess  the  vulnerability  of  AFVs  to  direct-fire 
weapons.  SQuASH’s  submodels  account  for  the  nonlinear  nature  of  many  types  of  interactions.  For  example, 
the  amount  of  spall  when  a  threat  defeats  an  armor  is  not  a  linear  function  of  the  hardness  of  the  armor,  above  a 
threshold  Brinell  hardness  value,  no  spall  is  produced.  The  precise  size,  shape,  and  velocity  of  spall  fragments, 
for  example,  are  best  modeled  as  a  stochastic  process.  Due  to  the  nonlinear  nature  of  fragment  penetration, 
expected-values  for  these  quantities  will  not  be  adequate  for  deteimining  damage  to  components.  SQuASH  out¬ 
put  metrics  include  field  observable  damage  to  internal  components  and  remaining  engineering  capability  of  the 
AFV.  The  primary  purpose  of  SQuASH  was,  and  remains,  to  provide  V/L  analysts  with  a  tool  to  make  live-fire 
pre-shot  predictions  and  to  perform  post-shot  analysis. 

The  original  stand-alone  version  of  SQuASH  was  coded  in  FORTRAN.  When  its  predictions  were  compared  to 
actual  LFT,  it  was  found  to  underpredict  component  damage  [Menne  1987;  Dively,  eL  al  1989;  Deitz,  et  al 
1989;  IDA  1994;  SURVICE  1994].  Significant  changes  to  SQuASH  have  taken  place  recently  in  two  areas: 
improvements  to  individual  algorithms  or  submodels,  and  inclusion  of  the  code  into  the  Modular  UNK™-based 
Vulnerability  Estimation  Suite  (MUVES).  The  latter  is  both  a  computer  code  for  modeling  and  simulating  V/L 
mechanisms  as  well  as  a  comprehensive  environment  for  conducting  V/L  analyses  [Hanes,  et  al  1991].  We 
have  reason  to  believe  that  these  changes  will  have  a  significant  impact  on  the  SQuASH-LFT  comparison  [Deitz 
and  Saucier  1995].  Consequently,  this  is  an  ideal  time  to  plan  a  verification  and  validation  of  the  model. 

Although  no  model  will  ever  totally  replace  testing,  it  is  equally  clear  that  there  will  never  be  enough  testing  to 
dispense  with  modeling  altogether.  A  properly  accredited  model  is  able  to  generalize  test  results  by  showing 
where  the  specific  test  result  lies  in  the  distribution  of  all  possible  outcomes.  The  proper  role  of  SQuASH,  then, 
is  to  supplement  testing  by  providing  insight  into  specific  damage  mechanisms.  Testing,  by  itself,  does  not  bring 
adequate  insight.  We  need  to  be  able  to  disentangle  the  individual  contributions  if  we  are  to  understand  issues  of 
vulnerability  reduction  and  lethality  enhancement. 

The  remainder  of  this  paper  presents  our  plan  for  a  comprehensive  verification  and  validation  of  the  SQuASH 
model  as  it  is  being  incorporated  into  MUVES.  First  we  present  the  formal  W&A  process,  followed  by  a  plan 
specific  to  the  SQuASH  model. 

THE  W&A  PROCESS 

The  basic  steps  in  the  W&A  process  [AR  5-11  1995;  DMSO  1995;  JTCG/ME  1993]  are  verification  of  both  the 
underlying  mathematical  model,  as  well  as  its  implementation  into  computer  code,  and  the  validation  of  the 
resulting  model,  as  depicted  in  Fig  1. 


Figure  1*  Role  of  Verification  and  Validation 

These  steps  are  fully  documented  and  given  to  an  ad-hoc  Methodology  Review  Committee  (MRC)  for  indepen¬ 
dent  review.  The  conclusions  and  recommendations  of  the  MRC  are  documented  and  in  turn  included  with  the 
other  documentation  to  form  a  complete  documentation  of  the  process.  The  complete  package  is  then  given  to  an 
agency  that  has  the  authority  to  make  a  decision  on  the  accreditation  of  the  model.  In  the  following  sections,  we 
elaborate  upon  each  of  these  basic  steps  in  the  W&A  process. 
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Verification 

Verification  begins  with  configuration  control,  that  is,  some  group  must  assume  the  responsibility  for  technical 
and  administrative  oversight  to  ensure  that  the  detailed  design  and  computer  source  code  are  documented  and 
tracked.  Logical  Verification  is  the  process  of  verifying  that  the  chosen  mathematical  model  is  appropriate  for 
the  purpose  at  hand  —  namely,  to  serve  as  an  approximation  of  the  real-world  phenomenon  which  is  simultane¬ 
ously  solvable  and  for  which  the  required  input  data  are  available.  Code  Verification  is  the  process  of  finding  a 
computer  model  to  implement  a  solution  of  the  underlying  mathematical  model.  Code  verification  assures  that 
the  code  is  an  accurate  translation  of  the  underlying  mathematical  model.  Code  verification  is  performed  in  a 
variety  of  ways  including  hand  calculations,  where  selected  elementary  cases  are  compared.  Also,  boundary 
conditions  are  tested  to  verify  that  the  model  properly  handles  input  that  lies  outside  the  intended  range.  Finally, 
documentation  is  checked  for  completeness  and  consistency  with  the  source  code  itself.  When  this  point  is 
reached,  sensitivity  analyses  are  used  to  ensure  that  the  model  output  is  sensitive  to  changes  in  model  input. 

Validation 

Validation  of  the  model  is  accomplished  through  a  variety  of  procedures.  Face  validation  is  performed  by  sub¬ 
ject-matter  experts  (SME)  who  check  the  model  for  reasonableness  based  upon  its  performance.  This  is  used  as 
a  point  of  departure  for  a  comprehensive  validation.  Model  assumptions  and  limitations  are  checked  to  assure 
that  they  are  appropriate  to  the  phenomena  they  represent.  Data  Audit  <Sc  Availability  assures  that  data  collection 
techniques  are  consistent  and  well-documented  and  that  the  data  requirements  for  the  model  are  realistic.  Sub¬ 
models  are  compared  to  results  obtained  under  laboratory  conditions.  The  overall  model  is  also  compared  to  lab¬ 
oratory  conditions,  where  possible,  and  also  to  LFT. 

Accreditation 

The  documentation  of  the  verification  and  validation  effort  will  be  turned  over  to  a  Methodology  Review  Com¬ 
mittee  (MRC).  This  can  be  an  ad-hoc  committee,  but  it  must  be  independent  of  the  M&S  developer.  The  finding 
of  the  MRC  will  be  documented  and  added  to  all  the  other  documentation.  Accreditation  is  an  official  determi¬ 
nation,  after  review  of  the  complete  documentation  of  the  process,  that  a  model  is  acceptable  for  a  specific  pur¬ 
pose.  The  level  of  V&V  required  for  accreditation  is  the  amount  sufficient  for  a  particular  application. 

W&A  OF  MUVES  SQUASH  MODEL 

In  abstract  form,  V/L  methodology  in  MUVES  SQuASH  can  be  considered  as  three  transformations  between 
four  distinct  spaces,  as  diagrammed  in  Fig  2. 


SQuASH 

Submodels 


Figure  2.  V/L  Taxonomy 

The  V/L  Taxonomy  Peitz  and  Ozolins  1988]  was  developed  to  conceptualize  this  process.  The  ellipses  in  this 
diagram  represent  four  distinct  spaces  or  Levels.  The  arrows  represent  mapping  operators  —  actually,  methodol¬ 
ogy  —  that  transform  information  at  one  Level  to  the  next.  The  following  diagram  depicts  this  process  in  more 
detail. 
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I 

Levd4]  Evaluate  Vehicle  (M,  F,  K)  LoF  [Vehicle  Utility  States] 

Figures.  FlowchartofMUVES Methodology 


In  the  case  of  direct-fire  weapons,  MUVES  begins  execution  at  Level  1,  as  this  is  typically  the  information  the 
analyst  provides.  However,  in  the  case  of  indirect-fire  weapons  —  an  example  is  artilleiy  (now  handled  by  the 
SAI^  approximation  method  within  MUVES)  —  execution  begins  at  Level  0.  At  each  threat-component  inter¬ 
action,  an  appropriate  Interaction  Module  (IM)  is  selected  in  order  to  compute  the  physical  interaction  with  the 
component.  However,  flie  damage  to  the  component  —  in  terms  of  whether  or  not  it  “survived”  the  encounter 
—  is  not  computed  at  this  point  in  the  process.  Rather,  all  of  the  information  to  compute  the  overall  damage  is 
placed  in  a  damage  packet,  which  is  placed  in  a  queue.  Only  after  all  threats  (main  penetrator,  pieces  of  the  bro¬ 
ken  penetrator,  behind-armor  debris  spall,  secondary  spall,  etc.)  have  been  “played”  is  the  damage  packet  pool 
evaluated.  Also  at  each  encounter,  the  threat  can  be  degraded  (mass  and  velocity  can  decrease),  it  can  be 
deflected  if  it  perforates  the  component,  or  it  can  ricochet  if  it  is  at  a  high  enough  obliquity  angle.  In  addition, 
the  threat  may  spawn  new  threats  by  generating  secondary  spall  or  pieces  due  to  penetrator  breakup.  TTie  origi¬ 
nal  (degraded)  threat,  plus  any  new  threats,  are  then  propagated  to  the  next  encounter.  It  is  clear  that,  in  general, 
there  is  not  a  single  undeviated  path  that  the  threat  traverses.  MUVES  handles  this  by  providing: 

•  dynamic  ray  tracing  (each  time  a  new  threat  is  spawned,  it  creates  a  new  ray  to  propagate  the  threat), 
and 

•  recursive  processing  (each  new  threat  is  put  on  the  same  footing  as  the  original  main  penetrator). 

Each  threat  continues  encountering  components  until  it  either  comes  to  rest  or  it  exits  the  vehicle.  After  all 
threats  have  been  processed  in  this  way,  the  queue  of  damage  packets  is  sorted  by  component.  The  appropriate 
Evaluation  Module  (EM)  is  selected  in  order  to  compute  the  target’s  functionality  or  degradation  (e.g.,  the  proba¬ 
bility  of  “killing”  [P*]  the  component).  Once  all  fire  critical  components  have  been  processed  (so  that  the  com¬ 
ponent  damage  states  are  known),  this  information  is  then  passed  along  to  the  fault  trees.  The  program  then 
computes  the  probability  of  “deactivating”  each  fault  tree  by  combining  all  the  individual  component  P^s. 
Component  P^s  represent  the  likelihood  that  a  particular  component  will  become  dysfunctional  given  an  interac¬ 
tion  wifli  a  threat.  From  this  combined  probability,  the  program  is  able  to  specify  the  system  damage  states.  It 
may  be  at  this  stage  (Level  3)  or  at  the  next  stage  (Level  4),  or  a  combination  of  the  two,  that  the  Degraded  Com¬ 
bat  Utility  (DCU)  in  the  Damage  Assessment  List  (DAL)  enters  the  computation.  This  depends  upon  how  the 
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DAL  is  constructed.  Typically,  the  DCU  values  are  specified  for  a  combination  of  critical  components  and  for 
critical  systems.  Finally,  knowing  the  system  damage  states,  the  probability  of  achieving  a  mobility,  firepower,  or 
catastrophic  ‘‘kill’'  can  be  computed.  MUVES  also  provides  many  post-processing  tools  for  analyzing  the  pro¬ 
gram  output,  but  these  are  not  shown  in  Fig  3. 

General  MUVES  Code 

MUVES  currently  consists  of  about  330,000  lines  of  source  code.  Approximately  85%  of  the  code  is  in  the 
nature  of  general  utility  and  not  specific  vulnerability  code  —  for  example,  error  handling,  or  memory  manage¬ 
ment.  These  packages  (i.e.,  logically  or  functionally  related  collections  of  subroutines)  have  been  well  tested  and 
are  in  good  shape.  The  majority  of  the  packages  in  MUVES  have  had  substantial  changes  recently  or  are  still 
evolving.  Examples  of  these  would  be  the  Distributions  of  Random  Numbers  and  the  Vector  Math  packages. 
The  final  group  of  MUVES  packages  are  those  that  are  not  yet  complete  and  include  the  Physical  Interaction 
package  and  the  SQuASH  Approximation  Method. 

There  are  four  procedures  that  are  being  employed  for  code  verification: 

•  Implementation:  Verify  that  the  code  is  an  accurate  translation  of  the  underlying  mathematical  model. 

•  Documentation:  Ensure  that  the  documentation  is  complete  and  consistent  with  the  code  itself. 

•  Test  Boundary  Conditions:  Verify  that  the  program  properly  handles  input  that  lies  outside  its  intended 
range  by  invoking  the  appropriate  warning  or  error  messages. 

•  Verify  Explicit  Calculations:  Compare  selected  cases  where  the  answer  is  known  or  can  be  calculated. 

SQuASH  Submodels 

SQuASH  submodels  cany  out  the  transformation  from  Level  1  to  Level  2,  and  will  undergo  verification  and  val¬ 
idation  as  stand-alone  programs.  The  following  is  the  list  of  submodels: 

•  Armor  Penetration 

—  Kinetic  Energy  Rounds 
—  Shaped  Charge  Jets 
—  Explosively  Formed  Penetrators 

•  Penetrator  Breakup 

•  Ammunition  Explosions 

•  Behind- Armor  Debris 

•  Fragment  Penetration,  Deflection,  and  Ricochet 

•  Component 

•  Crew  Incapacitation 

•  Hydraulic,  Fuel,  and  Water  Lines 

•  Wire  Bundles 

These  submodels  will  undergo  the  aforementioned  verification  procedures  as  well  as  laboratory  validation. 

SQuASH  Overall  Model  Validation 

Once  all  the  submodels  are  developed,  tested,  and  fully  assembled  in  the  SQuASH  infrastructure,  the  overall 
model  will  be  tested.  The  idea  here  is  to  test  the  data  flow  between  submodels,  check  that  each  submodel  has  the 
necessary  input,  check  for  possible  side  effects,  and  check  for  methodology  gaps  in  the  overall  structure.  We 
plan  to  use  both  a  bottom-up  and  top-down  approach  to  model  validation.  The  bottom-up  approach  will  consist 
of  a  validation  of  each  submodel  as  a  stand-alone  model.  The  top-down  approach  will  compare  the  complete 
model  to  LFT.  Once  these  have  been  completed,  we  will  be  at  Level  2  (see  Fig  2).  The  mappings  to  Level  3 
(Criticality  Analysis)  and  Level  4  (Operational  Analysis)  are  inputs  as  far  as  MUVES  is  concerned  and  are  dis¬ 
cussed  in  the  next  section. 

Verification  &  Vaiidation  of  SQuASH  inputs 

Ordinarily,  program  input  is  not  a  topic  for  program  verification  and  validation  —  the  cliche  is  “garbage  in, 
garbage  out.”  In  our  case,  we  need  to  concern  ourselves  with  V&V  of  input  since  it  is  a  major  part  of  MUVES. 
For  example,  the  target  description  consists  of  thousands  of  solids  and  is  very  complex.  Since  the  analysis  is 
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worthless  if  the  input  is  incorrect,  we  need  to  V&V  the  target  description.  The  following  areas  need  to  be  tested: 

•  Threats 

•  Target  Description 

•  Component  P* 

•  Degraded  States,  Fault  Trees,  and  Criticality  Analysis 

•  Performance  Metrics 

Component  will  most  likely  be  generated  off-line  with  a  separate  program.  The  results  from  this  program 
will  then  be  used  to  produce  tables  or  functions  that  will  go  into  SQuASH.  The  stand-alone  program  and  proce¬ 
dure  will  need  to  be  verified  and  validated. 

The  following  diagram  (Fig  4)  shows  all  the  critical  components  that  constitute  the  MlAl  Main  Armament. 


Main  Armament  Critical  Components  of  MlAl  Tank 


ID 

Description 

ID 

Description 

1 

Main  Gim  IVtbe 

10 

Gunner’s  Control  Hmidle 

2 

Main  Gun  Breech 

11 

Cable  1W200-9 

3 

Recoil  Mechanism 

12 

Cable  1W104 

4 

Recoil  Replenisher  tmd  Hose 

13 

Gtmner’s  Primary  Sight  -  Lower  Panel 

5 

Main  Gun  Thmntfins 

14 

Blasting  Machine 

6 

Ihrret  Networks  Box 

15 

Cable  1W105-9 

7 

Electric  Power  -  T\irret 

16 

Cable  1W107-9 

8 

Mimual  Elevation  Pump  Handle 

17 

Cable  1W108-9 

9 

Commander’s  Control  Handle 

18 

Main  Gtm  Safety  Switch 

Figure  4.  Fault  Tt«e  for  Main  Armament  of  MlAl  Tank 

If  any  of  these  critical  components  are  damaged,  then  the  vehicle  will  be  in  some  Degraded  State  (DS).  A  typi¬ 
cal  set  of  DS  for  an  AFV  is  shown  in  Fig  5. 
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Mobility  Subsystem 

Firepower  Subsystem 

Afo 

No  mobility  damage 

Fo 

No  firepower  damage 

A/j 

Slight  reduction  in  maximum  ^)eed 

Fi 

Loss  of  main  armament 

A/2 

Significant  reduction  in  maximum  speed 

Fz 

Unable  to  fire  on  the  move 

Afa 

Stop  after  time  t 

Fz 

Increased  time  to  fire 

M, 

Total  immobilization 

Fa 

Reduced  delivery  accuracy 

Mi 

Ml  and  Afa 

F, 

Loss  of  seconda^  armament 

M, 

M2  and  Afa 

Ft 

F  2  and  F  2 

Crew  Subsystem 

Fz 

Fj  and  F* 

Co 

0  crew  casualties 

Fz 

Fa  and  F* 

Cl 

I  crew  casualties 

F9 

F2  and  F3  and  Fa 

C2 

2  crew  casualties 

Fio 

Fj  and  F5 

Ca 

3  crew  casualties 

Fu 

F^andFs 

C4 

4  crew  casualties 

Fi2 

F4andF5 

Communication  Subsystem  { 

Fiz 

Fi  and  F3  and  F4  and  Fj 

Xo 

No  communication  damage 

Fia 

F2  and  F3  and  F3 

X, 

No  internal  communication 

Fi5 

F2  and  F4  and  F5 

No  external  communication  >  3{X)  feet 

Fi6 

F3  andF4  andFs 

X, 

No  external  communication 

Fn 

F 1  and  Fj  (total  loss  of  firepower) 

X4 

Xi  and  X2 

Ammunition  Subsystem 

Xs 

Xi  and  X2 

Xo 

No  ammunition  lost 

Acquisition  Subsystem  | 

Ki 

Bustle  ammunition  lost 

Ao 

No  acquisition  damage 

Kz 

Hull  ammunition  lost 

Ai 

Reduced  acquisition  capability 

Xz 

K\  and  K2 

A2 

Unable  to  acquire  while  moving 

Xa 

KkiU 

^3 

Ai  and  Ai 

Figures.  Enumeration  ofDegraded  States  for  an  AFV 

For  an  example  of  a  criticality  analysis,  see  [Ploskonka,  et.  at  1988].  Performance  metrics  include  mobility  and 
fire-power  loss-of-function  as  well  as  vehicle  catastrophic  “kill.” 

STATISTICAL  COMPARISON  TO  LFT 

Statistical  validation  methods  used  in  this  portion  of  the  SQuASH  validation  effort  include  two  procedures  that 
are  considered  to  be  judgmental  comparisons  and  an  additional  two  procedures  which  incorporate  hypothesis 
testing  techniques.  Judgmental  comparisons  are  widely  used.  They  include  graphical  analysis  and  the  compari¬ 
son  of  common  properties,  such  as  die  mean  and  variance  of  the  distribution  of  the  output.  They  are  easy  to  use 
and  quite  practical,  but  the  impact  of  errors  in  judgement  is  difficult  to  assess.  Hypothesis  testing  includes  good- 
ness-of-fit  tests,  analysis-of-variance  techniques,  and  nonparametric  ranking  methods;  they  allow  for  some 
degree  of  confidence  in  the  decision  but  (as  is  explained  later)  require  powerful  statistical  procedures. 

We  will  use  two  judgmental  comparisons  to  determine  whether  SQuASH  is  a  valid  predictor  of  component 
“kill.”  A  separate  analysis  will  be  conducted  for  each  live-fire  test,  although  results  can  be  combined  to  provide 
more  general  conclusions  concerning  the  model.  In  addition,  we  will  conduct  two  hypothesis  tests  over  all  live- 
fire  shots  simultaneously,  the  first  to  evaluate  SQuASH  as  a  predictor  of  component  “kill”  and  the  second  to 
evaluate  SQuASH  as  a  predictor  of  system  “kill.” 

The  initial  step  in  hypothesis  testing  is  to  state  a  null  hypothesis,  in  this  case  “the  simulation  model  is  valid.” 
Then  a  level  of  confidence  is  established,  often  95%;  and  a  particular  test  statistic  is  chosen.  TXvo  different  errors 
are  present  in  hypofliesis  testing.  The  first  is  called  a  Type  I  error  and  occurs  when  a  true  null  hypothesis  is 
rejected.  If  the  level  of  confidence  has  been  set  at  95%,  then  it  follows  that  the  probability  of  a  lype  I  error  is 
5%.  However,  in  simulation  model  validation,  a  Type  n  error  is  the  more  important  to  control;  this  occurs  when 
a  false  null  hypothesis  is  accepted.  No  level  of  confidence  is  pre-established  to  guard  against  accepting  an 
invalid  model;  but,  for  any  particular  statistical  test,  a  measure  of  the  protection  against  this  error  is  given  by  file 
power  of  the  test,  equal  to  the  probability  of  rejecting  the  null  hypothesis  when  it  is,  indeed,  false. 

Unfortunately,  there  is  a  tradeoff  between  the  two  error  types.  For  a  fixed  number  of  observations  (sample  size), 
as  the  level  of  confidence  is  increased  (lower  probability  of  a  Type  I  error),  the  power  of  the  test  is  decreased 
(higher  probability  of  a  Type  II  error).  A  Type  I  error  results  in  the  rejection  of  a  valid  model  —  unfortunate,  but 
not  as  potentially  dangerous  as  accepting  an  invalid  model,  the  Type  n  error.  Thus,  when  attempting  to  validate 
a  simulation  model  using  hypothesis  testing,  it  is  imperative  that  the  statistical  test  be  a  powerful  one. 

Box  Plots 

We  will  use  this  graphical  method  of  data  analysis  to  evaluate  SQuASH  as  a  predictor  of  component  vulnerabil¬ 
ity.  Level  2  metrics  will  be  compared  on  a  shot-by-shot  basis;  results  can  be  grouped  by  component  type  and 
threat  type. 

For  each  component  and  every  replication  of  the  model,  SC^uASH  outputs  the  probability  of  component  “Idll.” 
A  summary  display  of  the  distribution  of  this  random  variable  can  be  provided  by  the  box  plot  [Tukey  1977]. 
Using  this  method,  the  upper  and  lower  quartiles  (25th  and  75th  percentiles)  of  the  data  determine  the  sides  of  a 
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rectangle,  with  the  median  (50th  percentile)  portrayed  by  a  vertical  line  within  the  rectangle.  Dashed  lines 
extend  from  the  ends  of  the  box  to  the  upper  and  lower  adjacent  values  defined  as  the  largest  (smallest)  observa¬ 
tion  less  than  (greater  than)  or  equal  to  the  upper  (lower)  quartile  plus  (minus)  1.5  times  the  interquartile  range. 
Any  observation  falling  outside  the  range  of  the  two  adjacent  values  is  plotted  as  an  individual  point.  See  Fig  6 
for  an  example  of  a  box  plot. 
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Figure  6.  Example  of  a  Box  Plot 

The  box  plot  provides  a  quick  display  of  the  prominent  features  of  the  distribution  of  component  “kill”  probabil¬ 
ities.  Tbe  median  shows  die  location,  and  the  length  of  the  box  gives  an  indication  of  the  spread  of  the  bulk  of 
the  data.  Outside  points  allow  for  the  consideration  of  outliers.  The  overall  plot  indicates  the  degree  of  symme¬ 
try  within  the  distribution.  Box  plots  are  useful  in  situations  such  as  SQuASH  output  where  it  is  impractical  to 
display  all  the  details  of  the  distribution  (“kill”  probabilities  of  each  component  for  every  replication). 

SQuASH  now  outputs  box  plots  of  the  distribution  of  “kill”  probabilities  for  many  critical  components.  A  com¬ 
parison  with  the  live-fire  result  (“kill  or  no  kill”)  will  determine  whether  the  field  result  is  an  outside  point. 
Realizing  many  such  outside  points  will  reduce  confidence  in  the  validity  of  the  model. 

Institute  for  Defense  Analyses  (IDA)  Study 

We  will  use  this  comparison  method  of  data  analysis  to  evaluate  SQuASH  as  a  predictor  of  component  vulnera¬ 
bility.  Level  2  metrics  will  be  compared  on  a  shot-by-shot  basis;  results  can  be  grouped  by  component  type, 
threat  type,  and  damage  mechanism. 

This  method  was  proposed  in  a  briefing  prepared  by  the  Institute  for  Defense  Analyses  [TVimer,  Barr,  and 
Okamoto  1994].  From  each  live-fire  test,  it  requires  information  on  the  extent  of  component  damage,  as  well  as 
the  damage  mechanism  (penetrator,  spall,  etc.).  Damage  to  critical  components  is  grouped  into  five  categories: 

0. 8  <  SQuASH  P*<  1.0, 

0. 6  <  SQuASH  Pj<0.8, 

0. 4  <  SQuASH  Pit  <0.6, 

0. 2  <  SQuASH  Pt<0.4, 

0.0<  SQuASH  P*<0.2. 

PjjS  are  then  summed  within  the  categories  to  obtain  an  expected  number  of  damaged  components.  These  results 
are  presented  in  histogram  form,  along  with  the  actual  number  of  complete  and  partial  “kills”  obtained  from  the 
live-fire  test. 

Histograms  can  be  prepared  for  all  components  simultaneously,  or  the  results  can  be  grouped  by  system  compo¬ 
nents,  by  threat  types,  or  by  damage  mechanisms.  The  height  of  the  corresponding  histograms  indicates  whether 
the  model  underpredicted  or  overpredicted  in  each  simation.  Trends  toward  one  or  the  other  simation  reduce 
confidence  in  the  validity  of  the  model. 

Ordering  by  Probabilities  Test 

We  will  use  this  hypothesis  test  to  evaluate  SQuASH  as  a  predictor  of  component  vulnerability.  Level  2  metrics 
will  be  compared  over  all  live-fire  shots;  results  can  be  grouped  by  threat  type  and  damage  mechanism.  This  test 
is  also  applicable  to  validating  the  model’s  ability  to  predict  target  perforation. 

Let  P  =  (pi,P2>  •  •  • .  Pn)  a  vector  of  “kill”  probabilities  for  some  component  over  N  live-fire  shots.  These 
Pi’s  represent  the  mean  output  probability  over  M  replications  of  the  model.  The  null  hypothesis  is  that  the  Pi’s 
are  equal  to  the  true  probabilities  of  component  “kiH”  for  all  N  shots,  loosely  interpreted  as  “the  simulation 
model  is  valid.”  Hie  results  from  the  field  test  will  consist  of  an  iV-tuple,  the  values  of  which  are  either  1  or  0 
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(“kill  or  no  kill”).  There  are  2^  possible  combinations  of  this  A^-tuple,  and  its  distribution  can  be  obtained  by 
simply  considering  each  combination.  The  probability  of  any  given  -tuple  is  equal  to  the  product  of  the  proba¬ 
bilities  of  the  individual  elements  [p,-  if  the  element  is  equal  to  1,  or  (1  -  pi)  if  the  element  is  equal  to  0]. 

Once  the  probabilities  of  all  2^  combinations  have  been  evaluated,  they  are  ordered  and  form  the  cumulative  dis¬ 
tribution  of  P.  Assuming  a  level  of  confidence  of  95%,  those  outcomes  appearing  in  the  bottom  5%  of  the  distri¬ 
bution  are  considered  rare  events.  If  results  from  the  N  live-fire  shots  match  one  of  the  outcomes  in  the  tail  of 
the  distribution,  the  null  hypothesis  is  rejected.  It  is  important  to  remember  that  with  this  level  of  confidence,  we 
would  expect  even  a  valid  model  to  produce  1  rejection  in  every  20  tests.  Additional  statistical  tests  could  be 
used  to  combine  these  results  over  a  collection  of  components. 

Power  studies  of  this  hypothesis  test  demonstrated  better  power  than  that  of  similar  type  tests.  These  efforts, 
along  with  an  example  of  the  procedure,  are  provided  in  a  U.  S.  Army  Ballistic  Research  Laboratory  (BRL) 
report  [Webb  1989]. 

Combination  of  independent  Mann-Whitney  Tests 

We  will  use  this  hypothesis  test  to  evaluate  SQuASH  as  a  predictor  of  system  vulnerability.  Level  4  metrics,  val¬ 
ues  of  which  range  between  0  and  1,  will  be  compared  over  all  live-fire  shots;  results  can  be  grouped  by  threat 
type. 

Let  X  =  (xi,  X2,  •  •  • ,  X]^)  be  a  vector  of  inputs  to  SQuASH,  and  let  y  be  an  output  resulting  from  X.  Then  y  will 
take  on  a  value  for  each  replication  of  the  model.  Let  z  be  the  corresponding  value  from  the  live-fire  test  given 
the  same  input  vector.  In  general,  y  will  not  be  equal  to  z,  since  X  contains  only  a  finite  number  of  input  vari¬ 
ables  —  ostensibly,  the  most  relevant  ones.  The  purpose  of  SQuASH  is  to  mimic  the  real-world  process.  Thus, 
in  attempting  to  validate  it,  each  empirical  value  is  compared  to  the  corresponding  model  output  generated  under 
the  same  conditions;  that  is,  the  same  value  for  the  vector  X, 

Given  M  replications,  output  from  the  SQuASH  model  becomes  a  set  of  values  y\  *  •  • ,  for  each  set  of 
input  values  which  can  be  compared  with  (in  live-fire  testing)  a  single  corresponding  empirical  value  z.  Recall 
that  X  is  a  vector  of  most  (but  not  all)  of  the  relevant  input  variables.  Then  z,  given  X,  is  a  random  variable 
reflecting  the  random  error  due  to  the  exclusion  of  certain  factors  from  X.  Also  y,  of  course,  is  a  random  vari¬ 
able,  since  the  simulation  model  is  stochastic.  We  would  like  to  show  that  F(ylX),  the  conditional  distribution 
function  of  y,  is  equal  to  G(zlX),  the  conditional  distribution  function  of  z  for  all  0  <  y,  z  ^  1,  and  for  all  X. 

Considering  N  different  input  sets  (A^  live-fire  shots),  the  available  data  consist  of  N  observations 
(>^i»  ‘  )>  iyh  *  •  * » yi\  '  y  (ylfy  » yjv )  of  multivariate  random  variables,  where  the  y^s  for  any 

given  observation  share  a  common  distribution.  Ranking  yj ,  y/ ,  •  •  * ,  yf^,  and  Z/  for  each  i  provides  an  indication 
of  SQuASH  performance.  If  the  model  is  valid,  the  Z/  would  be  expected  to  fall  somewhere  in  the  middle  of 
such  a  ranking.  This  is  the  initial  step  in  a  procedure  known  as  the  Mann-Whitney  test,  a  particular  case  in 
which  one  of  the  random  variables,  namely  z,  ,  has  a  sample  size  of  one.  When  we  combine  N  of  these  tests,  we 
have  the  null  hypothesis  of  F(y!X)  =  G(ziX)  for  all  0  <  y,  z  ^  1  and  for  all  X,  which  we  can  interpret  as  “the 
simulation  model  is  valid.” 

Let  Ri  be  the  rank  of  z,-  in  the  ilh  observation  (y? ,  y?,  •  •  • ,  yf^,  z*);  thus,  R/  is  an  integer  between  1  and  Af  -h  1. 
Then  a  test  statistic  T  is  defined  as  the  sum  of  the  R^  ’s  over  all  N  observations.  Very  high  or  very  low  values  of 
T  will  cause  rejection  of  the  null  hypothesis,  since  this  would  indicate  that  the  model  has  a  tendency  to  underpre¬ 
dict  or  oveipredict.  The  theory  behind  the  Mann-Whitney  test  is  given  in  most  elementary  statistics  textbooks 
[Conover  1971],  and  the  properties  associated  with  the  combination  of  such  tests  have  been  documented  [Van 
Eltem  I960]. 

Power  studies  of  this  hypothesis  test  demonstrated  reasonable  power  against  appropriate  alternate  hypotheses. 
These  efforts,  along  with  an  example  of  the  procedure,  including  the  derivation  of  critical  values  for  the  test 
statistic,  are  provided  in  a  BRL  report  [Baker  and  Taylor  1985]. 

SUMMARY  AND  PLANS 

Modeling  has  been  attacked  recently  as  not  providing  a  credible  alternative  to  LFT.  But  a  properly  accredited 
model  has  real  savings  potential.  For  example,  M&S  of  a  radical  front-engine  design  for  the  M1A2  Tank 
demonstrated  its  vulnerability  to  certain  threats  —  later  confirmed  by  Desert  Storm  data  —  with  a  potential  sav¬ 
ings  of  $1(X)M.  There  is  a  role  for  both  modeling  and  testing;  it  is  not  a  question  of  one  or  the  other  but  rather  of 
finding  the  optimum  mix  of  the  two  that  is  most  cost-effective  to  the  Army. 
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The  key  to  establishing  the  credibility  of  SQuASH  is  a  comprehensive  and  properly  executed  W&A  plan.  The 
plan  that  we  have  outlined  in  this  paper  will  continue  to  be  developed  in  the  upcoming  months  as  we  compare 
the  model  to  laboratory  data  and  LFT. 
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